As you might (or might not) know, this summer, I have taken on mentoring of
a GSoC project by Simon Chopin (a.k.a.
laarmen) which goal is to bring
fedmsg, the Fedora Infrastructure message bus, to Debian. Most of the work I ll be talking about here is Simon s work, please send all the praise towards him (I can take the blame, though).
What is this about?
As the project proposal states, the idea is to provide Debian with a unified, real-time, and open mechanism of communication between its services. This communication bus would allow anyone, anywhere, to start consuming messages and reacting to events happening in Debian s infrastructure:
- trigger a test build on a git push to a source repository
- trigger automated testing (piuparts, lintian) as soon as an upload hits the archive
- get a desktop notification when a package you care about gets changed
-
When we told
upstream about our plan of adapting fedmsg to work on Debian, they were thrilled. And they have been
very supportive of the project.
How is the project going?
Are you excited? I know I m excited.
Well, the general idea was easy enough, but the task at hand is a challenge. First of all, fedmsg has
a lot of (smallish) dependencies, most of them new to Debian.
Thanks to Simon s work during the bonding period, and thanks to paultag s careful reviews, the first batch of packages (the first dependency level, comprising
kitchen,
bunch,
m2ext,
grapefruit,
txws,
txzmq and
stomper) is currently sitting in the NEW queue. The four remaining packages (
fabulous,
moksha.common,
moksha.hub and
fedmsg proper) are mostly ready, waiting in the Debian Python Module Team SVN repository for a review and sponsorship.
While we re waiting for the packages to trickle into Debian, Simon is not twiddling his thumbs. Work has taken place on a few fronts:
- Getting fedmsg integrated to
mentors.debian.net (actually sending the first messages from Debian s infrastructure)
- Writing a (stop-gap) wrapper to convert some of our mail broadcasts (debian-devel-changes, debian-bugs-dist) into fedmsg messages, until a more proper integration can be done
- Working on reliability of the message transport, following some concerns raised by DSA.
fedmsging mentors.debian.net
Package backports
mentors.debian.net was chosen because I m an admin and could do the integration quickly. That involved backporting the eleven aforementioned packages, plus
zeromq3 and
python-zmq (that only have TCP_KEEPALIVE on recent versions), to wheezy, as that s what the
mentors.d.n host is running. (Also,
python-zmq needs a new-ish
cython to build so I had to backport that too). Thankfully, those were no-changes backports, that were easily
scripted, using a pbuilder hook to allow the packages to depend on previously built packages.
I have made a wheezy package repository available
here. It s signed with my GnuPG key, ID
0xB8E5087766475AAF, which should be fairly well connected.
Code changes
After Simon s initial setup of debexpo (which is not an easy task), the code changes have been fairly simple (yes, this is just a proof of concept). You can see them on top of the live branch on
debexpo s sources. I finally had the time to make them live earlier this week, and
mentors.debian.net has been sending messages on Debian s fedmsg bus ever since.
Deployment
mentors.d.n sends its messages on five endpoints,
tcp://mentors.debian.net:3000 through
tcp://mentors.debian.net:3004. That is one endpoint per WSGI worker, plus one for the importer process(es). You can tap in directly, by following the instructions below.
debmessenger
Debmessenger is the
stop-gap email-to-fedmsg bridge that Simon is developing. The goal is to create some activity on the bus without disrupting or modifying any infrastructure service. It s written in
hy, and it leverages the existing Debian-related python modules to do its work, using inotify to react when a mail gets dropped in a Maildir.
Right now, it s supposed to understand changes mails (received from debian-devel-changes) and bugs mail (from debian-bugs-dist).
I ll work on deploying an instance of debmessenger this weekend, to create some more traffic on the bus.
Reliability of the bus
I suggested using fedmsg as this was something that
already existed, and that solved a problem identical to the one we wanted to tackle (open interconnection of a distribution s infrastructure services). Reusing a piece of infrastructure that already works in another distro means that we can share tools, share ideas, and come up with solutions that we might not have considered when working alone. The drawback is that we have to either adapt to the tool s idiosyncrasies, or to adapt the tool to our way of working.
One of the main points raised by DSA when the idea of using fedmsg was brought up, was that of reliability. Debian s infrastructure is spread in datacenters (and basements
) all over the world, and thus faces different challenges than Fedora s infrastructure, which is more tightly integrated. Therefore, we have to ensure that a
critical consumer (say, a buildd) doesn t miss any message it would need for its operation (say, that a package got accepted).
There has been work upstream, to ensure that fedmsg
doesn t lose messages, but we need to take extra steps to make sure that a given consumer can replay the messages it has missed, should the need arise. Simon has started
a discussion on the upstream mailing list, and is working on a
prototype replay mechanism. Obviously, we need to test scenarios of endpoints dropping off the grid, hence the work on getting some activity on the bus.
How can I take a look?
a.k.a. Another one rides the bus
(Picture Yves-Laurent Allaert, CC-By-SA v2.5 / GFDL v1.2 license)
So, the bus is pretty quiet right now, as only two kinds of events are triggering messages: a new upload to mentors.debian.net, and a new comment on a package there. Don t expect a lot of traffic. However, generating some traffic is easy enough: just login to mentors.d.n, pick
a package of mine (not much choice there), or a real package you want to review, and leave a comment. poof, a message appears.
For the lazy
Join
#debian-fedmsg on OFTC, and look for messages from the
debmsg bot.
Current example output:
01:30:25 <debmsg> debexpo.voms-api-java.upload (unsigned) --
02:03:16 <debmsg> debexpo.ocamlbricks.comment (unsigned) --
(definitely needs some work, but it s a start)
Listening in by yourself
You need to setup fedmsg. I have a repository of
wheezy packages and one of
sid packages, signed with my GnuPG key, ID
0xB8E5087766475AAF. You can add them to a file in /etc/apt/sources.list.d like this:
deb http://perso.crans.org/dandrimont/fedmsg-<sid wheezy>/ ./
Then, import my GnuPG key into apt (
apt-key add), update your sources (
apt-get update), and install fedmsg (
apt-get install python-fedmsg). The versions are
<< to anything real, so you should get the real thing as soon as it hits the archive.
Finally, in
/etc/fedmsg.d/endpoints.py, you can comment-out the Fedora entries, and add a Debian entry like this:
"debian": [
"tcp://fedmsg.olasd.eu:9940",
],
fedmsg.olasd.eu runs a fedmsg gateway connected to the
mentors.d.n endpoints, and thus forwards all the mentors messages. It ll be connected to debmessenger as soon as it s running too.
To actually see mesages, disable
validate_signatures in
/etc/fedmsg.d/ssl.py, setting it to
False. The Debian messages aren t signed yet (it s on the roadmap), and we don t ship the Fedora certificates so we can t authenticate their messages either.
Finally, you can run
fedmsg-tail --really-pretty in a terminal. As soon as there s some activity, you should get that kind of output (color omitted):
"i": 1,
"msg":
"version": "2.0.9-1.1",
"uploader": "Emmanuel Bourg <ebourg@apache.org>"
,
"topic": "org.debian.dev.debexpo.voms-api-java.upload",
"username": "expo",
"timestamp": 1373758221.491809
Enjoy real-time updates from your favorite piece of infrastructure!
What s next?
While Simon continues working on reliability, and gets started on message signing according to his schedule, I ll take a look at deploying the debmessenger bridge, and making the pretty-printer outputs useful for our topics. There will likely be some changes to the messages sent by debexpo, as we got some feedback from the upstream developers about making them work in the fedmsg tool ecosystem (
datanommer and
datagrepper come to mind).
You can tune in to Simon s weekly reports on the
soc-coordination list, and look at the discussions with upstream on the
fedora messaging-sig list. You can also catch us on IRC,
#debian-soc on OFTC. We re also hanging out on the upstream channel,
#fedora-apps on freenode.
What happened about the reproducible
builds effort this week:
Media coverage
Daniel Stender published an English translation of the article which originally
appeared in Linux Magazin in Admin Magazine.
Toolchain fixes
Fixes landed in the Debian archive:
this week(end) I was more in the "RC bug fix flow" than in the last weeks, &
here are the results:
